
United States Patent and Trademark Office 



UNITED STATES DEPARTMENT OF COMMERCE 
United States Patent and Trademark Office 
Address: COMMISSIONER FOR PATENTS 
P.O. Box 1450 

Alexandria, Virginia 223 13-1450 
www.uspto.gov 



APPLICATION NO. 


FILING DATE 


FIRST NAMED INVENTOR 


ATTORNEY DOCKET NO. 


CONFIRMATION NO. 


09/624,385 


07/27/2000 


Tatsushi Katayama 


35.G2626 


9572 



5514 7590 01/10/2005 

fitzpatrick cella harper & SCINTO 

30 ROCKEFELLER PLAZA 
NEW YORK, NY 10112 



EXAMINER 



WANG, JIN CHENG 



ART UNIT 



PAPER NUMBER 



2672 

DATE MAILED: 01/10/2005 



Please find below and/or attached an Office communication concerning this application or proceeding. 



PTO 90C (Rev 10/03) 



Office Action Summary 


Application No. 

09/624,385 


Applicant(s) 

KATAYAMA ET AL 


Examiner 

Jin-Cheng Wang 


Art Unit 

2672 





~ The MAILING DATE of this communication appears on the cover sheet with the correspondence address - 



Period for Reply 

A SHORTENED STATUTORY PERIOD FOR REPLY IS SET TO EXPIRE 3 MONTH (S) FROM 
THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 

• If the period for reply specified above is less than thirty (30) days, a reply within the statutory minimum of thirty (30) days will be considered timely. 

• If NO period for reply is specified above, the maximum statutory period will apply and will expire SIX (6) MONTHS from the mailing date of this communication. 

- Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 U.S.C. § 133). 
Any reply received by the Office later than three months after the mailing date of this communication, even if timely filed, may reduce any 
earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

I) S Responsive to communication(s) filed on 09/27/2004 . 

2a)S This action is FINAL. 2b)Q This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1 935 CD. 1 1 , 453 O.G. 21 3. 

Disposition of Claims 

4) ^ Claim(s) 1,2.5-8.11-13 and 26-29 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6) ^ Claim(s) 1.2.5-8.11-13 and 26-29 is/are rejected. 

7) D Claim(s) is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)D The drawing(s) filed on is/are: a)D accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 
Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1121(d). 

I I) D The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 

Priority under 35 U.S.C. § 119 

12)D Acknowledgment is made of a claim for foreign priority under 35 U.S.C. § 119(a)-(d) or (f). 
a)D All b)D Some * c)D None of: 

1 .□ Certified copies of the priority documents have been received. 

2. D Certified copies of the priority documents have been received in Application No. . 

3. D Copies of the certified copies of the priority documents have been received in this National Stage 

application from the International Bureau (PCT Rule 17.2(a)). 
* See the attached detailed Office action for a list of the certified copies not received. 



Attachment(s) 

1 ) □ Notice of References Cited (PTO-892) 4) Q Interview Summary (PTO-41 3) 

2) □ Notice of Draftsperson's Patent Drawing Review (PTO-948) Pa P er No(s)/Mail Date. . 

3) □ Information Disclosure Statement(s) (PTO-1449 or PTO/SB/08) 5 ) d Notice of Informal Patent Application (PTO-152) 

Paper No(s)/Mail Date . 6) □ Other: . 



U.S. Patent and Trademark Office 
PTOL-326 (Rev. 1-04) 



Office Action Summary 



Part of Paper No./Mail Date 20041202 



Application/Control Number: 09/624,385 . Page 2 
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DETAILED ACTION 
Response to Arguments 

Applicant's arguments filed Sept. 27, 2004 have been fully considered but they are not 
persuasive. As addressed below, Xiong, Shum and Teo teach the claim limitations. 

Applicant argues in essence with respect to the claim 1 and similar claims that Teo does 
not teach automatically setting the mapping mode. However, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that automatically provides feedback to the computer 
system when the pair-wise objective function is not desirable for a poor selection of the 
projection viewing plane and the resulting panoramas will have imperfectly aligned images that 
give shadow or ghosting effects due to a poor selection of the projection viewing plane and the 
resulting panoramas will have imperfectly aligned images that give shadow or ghosting effects. 
Xiong further teaches a human interaction being present throughout the image synthesis process 
to provide feedback to the computer system in all the nonlinear optimizations to let users monitor 
the progress of the system and allow them to intervene when necessary. 

Teo teaches that the viewing software can automatically invert the projection by 
automatically straightening out the portion of the panoramic image being displayed and thereby 
automatically setting the mapping mode by the viewer software (column 7-8). Furthermore, the 
claim limitation of "automatically setting the mapping mode" does not necessarily imply 
automatically setting the mapping surface. Teo teaches the image processor for automatically 
transforming the first digital panoramic image into a second digital panoramic image 
corresponding to the projection of the scene onto the second surface geometry and therefore 
automatically setting the mapping mode of mapping the images to the selected surface geometry 
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(column 7). Teo teaches in column 10 the mapping mode of mapping onto the planar surfaces 
such as mapping the panorama images onto a cube and the mapping mode of mapping the 
panorama images onto polyhedral surfaces to form cylindrical panorama images. Teo teaches 
that a user can change a mapping mode ( column 1L lines 1-9 ) by adjusting the shape of the 
mapping surfaces so that a given cylindrical panoramic image can be transformed to produce a 
modified panoramic image so that the image processor can automatically transform the 
panoramic images (Fig. 9). Teo teaches that a user can modify the surface geometry by 
manipulating, e.g., adjusting the scale of the polyhedral surface, rotating the surface, re- 
positioning edges of the polyhedral surfaces upon which the scene is to be projected, relative to 
the initial panoramic image, to form a desired surface, wherein the desired surface is distinct 
from the initial surface and modifying the initial panoramic image by mapping points on the 
desired surface to corresponding points on the initial surface to produce a modified panoramic 
image corresponding to projection of the scene onto the desired surface to which the panoramic 
image is mapped. Teo therefore teaches automatically setting a mapping mode or changing the 
plane for mapping the panoramic image by the computer through a user interface. 

With regards to the newly added claim limitations, Teo teaches horizontal placement of 
image by projecting a scene onto a cube so that horizontal lines of the scene appear horizontal in 
the panoramic image, and vertical placement of image by projecting the panoramic image on the 
vertical cylinder or by rotating the polyhedral surface relative to a cylindrical panoramic image 
and making adjustments to the sides and edges of the polyhedral surface so that the placement of 
images appear vertically on the mapping surface. Teo teaches the user interface enabling the user 
to rotate the polyhedral surface relative to a cylindrical panoramic image and making 
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adjustments to the sides and edges of the polyhedral surface so that the placement of images 
appear vertically on the mapping surface. 

Shum teaches a plurality of mapping modes such as zooming and panning the screen 
images of the interior scene shown in the panoramic view of Fig. 10A wherein the mapping 
surfaces within panoramas are different (See Figs. 1 1 A and 1 IB and column 23-24) and 
changing the type of panoramas such as the spherical panoramas (Figs. 15A-15B) and the planar 
panoramas (Figs. 16A-16B). Shum teaches a setting step of zooming/panning the panorama and 
setting the spherical panoramas (column 23-24) by the interactive user interface so that the 
mapping mode changes in which each resulting panorama has different mapping surfaces 
(column 23-24). Shum further teaches determining the constraints governing the mapping mode 
of the mapping onto the projection plane wherein the constraints incorporates the predetermined 
panoramic view of a 3D scene or the predetermined camera orientation matrix by the governing 
linear systems of equations. Shum further teaches generating a warning message to indicate that 
the constraint equations are unsolvable in a case in which the synthesized image is not aligning 
with any portion of the projected model in response to the change in the geometric constraints 
determined by the projection plane or in response to the change in the panoramic view. The 
geometric constraints change due to the change in the projection plane from a planar panorama 
to a spherical panorama wherein the planes within the spherical panorama are changed to 
spherical planes and the panorama view can be changed as well by zooming/panning panorama 
and/or selecting a portion of the panorama. Therefore, un-solvability of the linear equations 
occurs in the process of changing the mapping mode because the change in the mapping mode or 
the change in the geometric surface also changes the geometric constraints governing the linear 
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equations. When the linear systems of equations are not solvable, a user can switch the mapping 
plane orientation or changing the mode of projection surface to meet the constraint equations 
requirements wherein the plane normals have changed. Moreover, the projection space can be in 
the form of any shape such as spherical or planar and by changing the camera's point of view 
and/or the panorama view and the coordinates and directions of the plane normals, the projection 
space is effectively changed and accordingly the mapping mode is changed ( column 13-24 ). 

Applicant argues that no initial mapping mode exists in Shum to which a change applies. 
However, Shum teaches changing the mapping mode by zooming and panning the screen images 
of the interior scene shown in the initial panoramic view of Fig. 10A (See Figs. 1 1 A and 1 IB 
and column 23-24) and changing the type of panoramas such as the spherical panoramas (Figs. 
15A-15B) and the planar panoramas (Figs. 16A-16B). Shum teaches a setting step of 
zooming/panning the panorama and setting the spherical panoramas (column 23-24) by the 
interactive user interface so that the mapping mode changes wherein each resulting panorama 
has different mapping surfaces in accordance with the user input commands (column 23-24) 
wherein the panorama mapping surfaces change as a result of the zooming and panning. 

Claim Rejections - 35 USC §103 
1. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 
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2. Claims 1-2, 5-8, 11-13, and 26-29 are rejected under 35 U.S.C. 103(a) as being 
unpatentable over Xiong et al. U.S. Patent No. 6,434,265 (hereinafter Xiong), and further in view 
of Shum et al. U.S. Patent No. 6,271,855 (hereinafter Shum) and Teo U.S. Patent No. 6,246,413 
(Teo). 

3. Claim 1: 

(a) Xiong teaches an image synthesis method comprising: 

An input step, of inputting a plurality of image data (e.g., column 9, lines 20-30); 

A placement information generating step, of generating placement information 

determined by a placement order of all images inputted in the input step (e.g., column 16, lines 

35-45); 

A placement information obtaining step of obtaining placement information about a 
plurality of images in which adjacent images have a common subject region (e.g., in column 4, 
lines 5-40, Xiong teaches a method for constructing a panorama from rectilinear images in 3D 
through projective registration and calibration including: (1) the projective registrations of 
overlapping images, (2) calibration and global optimization of these images, a self calibration in 
which 2D image planes are positioned as 3D planes in space)', 

A setting step of setting one mapping mode out of a plurality of mapping modes each 
corresponding to a different mapping surface in accordance with the obtained placement 
information (e.g., in column 8, lines 18-58 of Xiong, it is stated that "overlapping photographs 
are analyzed to determine what orientation the photographs were taken in order to establish a 
common ground for subsequent operations and the panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
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viewing plane for viewing". The Xiong discloses some typical geometry on which panoramas are 
formed; In column 8, lines 18-58, Xiong further discloses thai panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen viewing 
plane such as cubic, polyhedral, cylindrical and spherical geometries); and 

A synthesis step of combining the plurality of images by using the mapping mode set in 
the setting step (e.g., in column 4, lines 5-40 of Xiong, it is stated that the composing or blending 
in which images are ready to be re-projected to a 3D environment map with pixels in overlap 
regions being composed from multiple; In column 8, lines 18-58, Xiong further teaches that 
overlapping photographs are analyzed to determine what orientation the photographs were, 
taken in order to establish a common ground for subsequent operations and the panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection of 
the panorama onto a chosen viewing plane for viewing. The Xiong further discloses some typical 
geometry on which panoramas are formed) ', 

A changing step, of changing the mapping mode (In column 4, lines 40-50, it is stated 
that "the projection module may be controlled through the user interface 230 as well, to allow a 
user to select what geometry will be projected onto". Therefore, Xiong teaches a changing step 
through the user interface 230 by selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface). 

- The Examiner interprets "a placement information obtaining step of obtaining placement 
information about a plurality of images in which adjacent images have a common subject 
region" as an automatic registration and calibration step of registering the overlapping 
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images and capturing common overlapping areas between overlapping images and 
minimizing the average squared pixel intensity difference with respect to certain 
transformation parameters. 
- The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
mapping modes each corresponding to a different mapping surface in accordance with 
said obtained placement information. 

(b) However, Xiong is silent to "a setting step, of automatically setting one mapping 
mode out of a plurality of mapping modes each corresponding to a different mapping surface by 
computations involving the horizontal and vertical placement of images." 
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(c) Teo teaches the claim limitation of a setting step, of automatically setting one 
mapping mode out of a plurality of mapping modes each corresponding to a different mapping 
surface by computations involving the horizontal and vertical placement of images {Teo teaches 
that the viewing software can automatically invert the projection by automatically straightening 
out the portion of the panoramic image being displayed and thereby automatically setting the 
mapping mode by the viewer software in column 7-8. Furthermore, Teo teaches the image 
processor for automatically transforming the first digital panoramic image into a second digital 
panoramic image corresponding to the projection of the scene onto the second surface geometry 
and therefore automatically setting the mapping mode of mapping the images to the selected 
surface geometry in column 7. Teo teaches in column 10 the mapping mode of mapping onto the 
planar surfaces such as mapping the panorama images onto a cube and the mapping mode of 
mapping the panorama images onto polyhedral surfaces to form cylindrical panorama images. 
Teo teaches that a user can change a mapping mode, see column 77. lines 1-9. by adjusting the 
shape of the mapping surfaces so that a given cylindrical panoramic image can be transformed 
to produce a modified panoramic image. See also Fig. 9. Teo teaches that a user can modify the 
surface geometry by manipulating, e.g., adjusting the scale of the polyhedral surface, rotating 
the surface, re-positioning edges of the polyhedral surfaces upon which the scene is to be 
projected, relative to the initial panoramic image, to form a desired surface, wherein the desired 
surface is distinct from the initial surface and modifying the initial panoramic image by mapping 
points on the desired surface to corresponding points on the initial surface to produce a modified 
panoramic image corresponding to projection of the scene onto the desired surface. Teo 
therefore teaches automatically setting a mapping mode or changing the plane for mapping the 
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panoramic image by the computer through a user interface. Finally, Teo teaches horizontal 
placement of image by projecting a scene onto a cube so that horizontal lines of the scene 
appear horizontal in the panoramic image and vertical placement of image by projecting the 
panoramic image on the vertical cylinder or by rotating the polyhedral surface relative to a 
cylindrical panoramic image and making adjustments to the sides and edges of the polyhedral 
surface so that the placement of images appear vertically on the mapping surface. Teo teaches 
the user interface enabling the user to rotate the polyhedral surface relative to a cylindrical 
panoramic image and making adjustments to the sides and edges of the polyhedral surface so 
that the placement of images appear vertically on the mapping surface). 

(d) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Teo's setting step because Xiong also teaches a user interface for setting a mapping surface by 
the computer and therefore Xiong suggests automatically setting a mapping mode. 

(e) One having the ordinary skill in the art would have been motivated to do this because 
setting a mapping mode out from a plurality of mapping mode serves for the purpose of 
eliminating visible artifacts (Teo column 10). 

(f) However, Xiong and Teo are silent to issuing warning wherein the warning being 
issued in a case in which the synthesized image exceeds a predetermined angle of view when a 
cylindrical mapping mode is changed to a plane mapping mode in the claim limitation of "a 
generating step, of issuing , when an image formed by changing the mapping mode in the changing 
step does not comply with a predetermined condition set in accordance with the mapping mode, a 
warning and generating a synthesized image in accordance with the predetermined condition, the 
warning being issued in a case in which the synthesized image exceeds a predetermined angle of 
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view when a cylindrical mapping mode is changed to a plane mapping mode" (i.e., generation of a 
warning message). 

(g) Shum implicitly teaches issuing warning wherein the warning being issued in a case 
in which the synthesized image exceeds a predetermined angle of view when a cylindrical 
mapping mode is changed to a plane mapping mode in the claim limitation of "a generating step, 
of issuing , when an image formed by changing the mapping mode in the changing step does not 
comply with a predetermined condition set in accordance with the mapping mode, a warning and 
generating a synthesized image in accordance with the predetermined condition, the warning 
being issued in a case in which the synthesized image exceeds a predetermined angle of view 
when a cylindrical mapping mode is changed to a planar mapping mode while simultaneously 
generating and displaying the synthesized image within the predetermined range of viewing 
angle" (e.g., Shum teaches a plurality of mapping modes such as zooming and panning screen 
images of the interior scene shown in the panoramic view of Fig. J OA. See Figs. 11A and 11B 
and column 23-24. Shum also teaches the type of panoramas such as the spherical panoramas in 
Figs. 15A-15B and the planar panoramas in Figs. 16A-16B. Shum teaches a setting step of 
zooming/panning the panorama and setting the spherical panoramas in column 23-24 by the 
interactive user interface so that the mapping mode changes wherein each resulting panorama 
has different mapping surfaces in accordance with the user input commands. Shum further 
teaches determining the constraints governing the mapping mode of the mapping onto the 
projection plane wherein the constraints incorporates the predetermined angle of view such as 
the panoramic view of a 3D scene or the predetermined camera orientation matrix by the 
governing linear systems of equations. Shum further teaches generating a warning message to 
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indicate that the constraint equations are unsolvable in a case in which the synthesized image is 
not aligning with any portion of the projected model when the projection plane is changed or 
when the angle of view is changed in response to the user commands, e.g., the projection plane is 
changed from a planar panorama to a spherical panorama wherein the planes within the 
spherical panorama are changed to spherical planes and the panorama view can be changed as 
well by zooming/panning panorama and/or selecting a portion of the panorama. When the linear 
systems of equations are not solvable, a user can switch the mapping plane orientation or 
changing the mode of projection surface to meet the constraint equations requirements by 
changing the plane normals. Moreover, the projection space can be in the form of any shape 
such as spherical or planar and by changing the camera 's point of view and/or the panorama 
view and the coordinates and directions of the plane normals, the projection space is effectively t 
changed and thereby the mapping mode is changed. See column 13-24. ) 

(h) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Shum's warning message generation step because Xiong suggests a generating step of generating 
a synthesized image in accordance with the predetermined condition (Xiong column 3, lines 35- 
55; column 17, lines 15-67; column 18, lines 1-4). Moreover, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that provides feedback to the computer system such as 
issuing warning messages on the computer monitor 218 when the pair-wise objective function is 
not desirable for a poor selection of the projection viewing plane (and the resulting panoramas 
will have imperfectly aligned images that give shadow or ghosting effects) due to a poor 
selection of the projection viewing plane and the resulting panoramas will have imperfectly 
aligned images that give shadow or ghosting effects. Xiong also points to a seamless multi- 
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resolution average blending method that would result in an absent of shadow effects (column 14, 
lines 1-45). Xiong further teaches a human interaction being present throughout the image 
synthesis process to provide feedback to the computer system in all the nonlinear optimizations 
to let users monitor the progress of the system and allow them to intervene when necessary and 
therefore the claimed limitation suggests an obvious modification of Xiong because providing 
feedback to users is similar to issuing a message to users in a user interaction with the nonlinear 
optimizations when the changing step does not comply with a predetermined condition set or 
user- selected parameter set. 

(i) One having the ordinary skill in the art would have been motivated to do this because 
this would have provided the user the informative message such as issuing a warning message 
through the user interaction when convergence to the predetermined condition setting cannot be 
obtained in the image synthesis (Xiong column 17, lines 15-67, column 18, lines 1-4) for non- 
solvable optimization problems (Shum column 18, lines 38-67). 

Claim 2: 

Claim 2 recites all the limitations of claim 1 and adds the limitation of "a focal length 
obtaining step " Xiong teaches finding projective parameters such as 3D rotation parameters 
(pan, tilt roll), center of projection of images, ratio of focal lengths, and the like (column 10, 
lines 9-28). Xiong implicitly teaches finding the camera internal parameters vector of which the 
focal length is a component (column 1 1, lines 15-42). 



Claim 5: 
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The claim 5 recites all the limitations of claim 1 and adds the limitation of "a displaying 
step of displaying a cuttable rectangular region " Xiong teaches how to align images more 
precisely by changing the coordinates for positioning an image. Xiong further teaches placing the 
images 1210 at selected tangents on the viewing sphere 1220 (figure 12, and column 17, lines 
12-65). 

4. Claim 6: 

The claim 6 encompasses the same scope of invention as that of claim 1 except additional 
claimed limitation of "an image synthesis apparatus" and "a displaying step of displaying a 
cuttable rectangular region". However, Xiong further discloses in figure 3 an apparatus for image 
synthesis. Xiong teaches how to align images more precisely by changing the coordinates for 
positioning an image. Xiong further teaches placing the images 1210 at selected tangents on the 
viewing sphere 1220 (figure 12, and column 17, lines 12-65). 

Claim 7: 

The claim 7 recites all the limitations of claim 6 and adds the limitation of "a focal 
length obtaining step." The Xiong reference teaches finding projective parameters such as 3D 
rotation parameters (pan, tilt roll), center of projection of images, ratio of focal lengths, and the 
like (column 10, lines 9-28). Xiong implicitly teaches finding the camera internal parameters 
vector in his image synthesis apparatus of figure 3 because the focal length is a component of 
that vector (column 1 1, lines 15-42). 

Claim 8: 
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The claim 8 recites all the limitations of claim 6 and adds the limitation of "a changing 
step of changing the mapping mode." Xiong teaches that panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane such as cubic, polyhedral, cylindrical and spherical geometries (column 8, lines 
18-58). 



5. Claim 11: 

The claim 1 1 encompasses the same scope of invention as that of claim 1 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method. However, Xiong further discloses the claimed limitation 
of a computer-readable storage medium having a program for implementing image synthesis 
method (e.g., in column 3, lines 54-57, it is stated "a program residing in system memory 220 
which stores output data and other data"). 

Claim 12: 

The claim 12 recites all the limitations of claim 1 1 and adds "a focal length obtaining 
step." Xiong teaches finding projective parameters such as 3D rotation parameters (pan, tilt roll), 
center of projection of images, ratio of focal lengths, and the like (column 10, lines 9-28). Xiong 
implicitly teaches finding the camera internal parameters vector of which the focal length is a 
component (column 11, lines 15-42). 

Claim 13: 

The claim 13 recites all the limitations of claim 1 1 and adds the limitation of "a changing 
step of changing the mapping mode." Xiong teaches that panorama is constructed on a particular 
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geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
viewing plane such as one of the cubic, polyhedral, cylindrical and spherical geometries (column 
8, lines 18-58). 

6. Claim 26: 

(a) Xiong teaches an image synthesis method comprising: 

An input step, of inputting a plurality of image data (e.g., column 9, lines 20-30); 

A placement information generating step, of generating placement information 

determined by a placement order of all images inputted in the input step (e.g., column 16, lines 

35-45); 

A placement information obtaining step of obtaining placement information about a 
plurality of images in which adjacent images have a common subject region (e.g., in column 4, 
lines 5-40, Xiong teaches a method for constructing a panorama from rectilinear images in 3D 
through projective registration and calibration including: (1) the projective registrations of 
overlapping images, (2) calibration and global optimization of these images, a self calibration in 
which 2D image planes are positioned as 3D planes in space); 

A setting step of setting one mapping mode out of a plurality of mapping modes each 
corresponding to a different mapping surface in accordance with the obtained placement 
information (e.g., in column 8, lines 18-58 of Xiong, it is stated that "overlapping photographs 
are analyzed to determine what orientation the photographs were taken in order to establish a 
common ground for subsequent operations and the panorama is constructed on a particular 
geometry that will best facilitate the rendering of the projection of the panorama onto a chosen 
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viewing plane for viewing". The Xiong discloses some typical geometry on which panoramas are 
formed; In column 8, lines 18-58, Xiong further discloses that panorama is constructed on a 
particular geometry that will best facilitate the rendering of the projection onto a chosen viewing 
plane such as cubic, polyhedral, cylindrical and spherical geometries); and 

A synthesis step of combining the plurality of images by using the mapping mode set in 
the setting step (e.g., in column 4, lines 5-40 of Xiong, it is stated that the composing or blending 
in which images are ready to be re-projected to a 3D environment map with pixels in overlap 
regions being composed from multiple; In column 8, lines 18-58, Xiong further teaches that 
overlapping photographs are analyzed to determine what orientation the photographs were taken 
in order to establish a common ground for subsequent operations and the panorama is 
constructed on a particular geometry that will best facilitate the rendering of the projection of the 
panorama onto a chosen viewing plane for viewing. The Xiong further discloses some typical 
geometry on which panoramas are formed); 

- The Examiner interprets "a placement information obtaining step of obtaining placement 
information about a plurality of images in which adjacent images have a common subject 
region" as an automatic registration and calibration step of registering the overlapping 
images and capturing common overlapping areas between overlapping images and 
minimizing the average squared pixel intensity difference with respect to certain 
transformation parameters. 

The Examiner interprets the mapping mode as mapping images onto a geometric surface 
such as a planar or a cylindrical surface (Applicant's specification, page 1, lines 20-25). 
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Accordingly, the Examiner interprets the setting step of setting a mapping mode as the 
selecting step of selecting a geometric surface. In column 8, lines 18-58, Xiong discloses 
that panorama is constructed on a particular geometry that will best facilitate the 
rendering of the projection onto a chosen viewing plane (a geometric surface) such as 
cubic, polyhedral, cylindrical and spherical geometries. In column 4, lines 40-50, it is 
stated that "the projection module may be controlled through the user interface 230 as 
well, to allow a user to select what geometry will be projected onto". Therefore, Xiong 
teaches a selecting step of selecting a geometric surface out of a plurality of geometric 
surfaces each corresponding to a different mapping surface in accordance with the 
obtained placement information. As applied to the present application, Xiong fulfills the 
claimed limitation of a setting step of setting one mapping mode out of a plurality of 
mapping modes each corresponding to a different mapping surface in accordance with * 
said obtained placement information. 

(b) However, Xiong does not implicitly teach "a setting step, of automatically setting one 
mapping mode out of a plurality of mapping modes each corresponding to a different mapping 
surface in accordance with the obtained placement information." 

(c) Teo teaches the claim limitation of a setting step, of automatically setting one 
mapping mode out of a plurality of mapping modes each corresponding to a different mapping 
surface in accordance with the obtained placement information (Teo teaches manipulating 
(adjusting the scale of the polyhedral surface, rotating the surface, re-positioning edges of the 
polyhedral surfaces) a polyhedral surface upon which the scene is to be projected, relative to the 
initial panoramic image, to form a desired surface, wherein the desired surface is distinct from 
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the initial surface and modifying the initial panoramic image by mapping points on the desired 
surface to corresponding points on the initial surface to produce a modified panoramic image 
corresponding to projection of the scene onto the desired surface. Teo therefore teaches 
automatically setting a mapping mode by the computer through a user interface); 

(d) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Teo's automatically setting step because Xiong teaches a user interface for setting a mapping 
surface by the computer automatically and therefore Xiong suggests automatically setting a 
mapping mode. 

(e) One having the ordinary skill in the art would have been motivated to do this because 
setting a mapping mode out from a plurality of mapping mode serves for the purpose of 
eliminating visible artifacts (Teo column 10). 

(f) However, Xiong and Teo are silent to issuing warning wherein the warning being 
issued in a case in which the synthesized image exceeds a predetermined angle of view when a 
cylindrical mapping mode is changed to a plane mapping mode. 

(g) Shum implicitly teaches issuing warning wherein the warning being issued in a case 
in which the synthesized image exceeds a predetermined angle of view when a cylindrical 
mapping mode is changed to a plane mapping mode (e.g., Shum teaches determining the 
constraints governing the mapping mode of the mapping onto the projection plane wherein the 
constraints incorporates the predetermined angle of view or the predetermined camera 
orientation matrix by the governing linear systems of equations. Shum further teaches generating 
a warning message to indicate that the constraint equations are unsolvable in a case in which the 
synthesized image is not aligning with any portion of the projected model when the projection 
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plane is changed or when the angle of view is changed, e.g., the projection plane is changed from 
a cylindrical plane to a spherical plane or the angle of view has changed. When the linear 
systems of equations are not solvable, a user can switch the mapping mode to meet the constraint 
equations requirements by changing the plane normals. Moreover, the projection space can be in 
the form of any shape such as spherical or cylindrical and by changing the camera's point of 
view and the coordinates and directions of the plane normals, the projection space is effectively 
changed and thereby the mapping mode is changed. See column 13-22). . 

(h) It would have been obvious to one of ordinary skill in the art to have incorporated the 
Shum's warning message generation step because Xiong suggests a generating step of generating 
a synthesized image in accordance with the predetermined condition (Xiong column 3, lines 35- 
55; column 17, lines 15-67; column 18, lines 1-4). Moreover, Xiong teaches in figures 2 and 3 a 
user interface and a global optimization that provides feedback to the computer system such as 
issuing warning messages on the computer monitor 218 when the pair-wise objective function is 
not desirable for a poor selection of the projection viewing plane (and the resulting panoramas 
will have imperfectly aligned images that give shadow or ghosting effects) due to a poor 
selection of the projection viewing plane and the resulting panoramas will have imperfectly 
aligned images that give shadow or ghosting effects . Xiong also points to a seamless multi- 
resolution average blending method that would result in an absent of shadow effects (column 14, 
lines 1-45). Xiong further teaches a human interaction being present throughout the image 
synthesis process to provide feedback to the computer system in all the nonlinear optimizations 
to let users monitor the progress of the system and allow them to intervene when necessary and 
therefore the claimed limitation suggests an obvious modification of Xiong because providing 
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feedback to users is similar to issuing a message to users in a user interaction with the nonlinear 
optimizations when the changing step does not comply with a predetermined condition set or 
user-selected parameter set. 

(i) One having the ordinary skill in the art would have been motivated to do this because 
this would have provided the user the informative message such as issuing a warning message 
through the user interaction when convergence to the predetermined condition setting cannot be 
obtained in the image synthesis (Xiong column 17, lines 15-67, column 18, lines 1-4) for non- 
solvable optimization problems (Shum column 18, lines 38-67). 



7. Claim 27: 

The claim 27 encompasses the same scope of invention as that of the claim 26 except 
additional claim limitation of "a displaying step of displaying a cuttable rectangular region." 
Xiong teaches how to align images more precisely by changing the coordinates for positioning 
an image. Xiong further teaches placing the images 1210 at selected tangents on the viewing 
sphere 1220 (figure 12, and column 17, lines 12-65). 

8. Claim 28: 

The claim 28 encompasses the same scope of invention as that of claim 26 except 
additional claimed limitation of a computer-readable storage medium having a program for 
implementing image synthesis method. However, Xiong further discloses the claimed limitation 
of a computer-readable storage medium having a program for implementing image synthesis 
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method (e.g., in column 3, lines 54-57, it is stated "a program residing in system memory 220 
which stores output data and other data"). 
Claim 29: 

The claim 29 encompasses the same scope of invention as that of the claim 1. The claim 
29 is subject to the same rationale of rejection set forth in the claim 1. 



Conclusion 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time 
policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1. 136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the mailing 
date of this final action. 

Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Jin-Cheng Wang whose telephone number is (703) 605-1213. 
The examiner can normally be reached on 8:00 - 6:30 (Mon-Thu). 
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If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Mike Razavi can be reached on (703) 305-4713. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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